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1. BACKGROUND 


Label Identification from Statistical Tabulation (LIST) Is a semi automated 
labeling procedure developed to integrate analyst- Interpreter (AI) Information 
and Land Satellite (Landsat) spectral data Into a consistent technology for 
labeling picture elements (pixels) as either small grain or nonsmall grain. 
This procedure Is designed for labeling grains through the use of postharvest 
Landsat data. It requires four acquisitions of satellite data during the 
groMing season. In this procedure, the AI Is required to answer questions 
about the se^nent and the Individual pixels which relate to simple properties 
that discriminate small -grain pixels from other pixels. The AI responses to 
these questions are confined with spectral, agricultural, and meteorological 
data to obtain keys or features for discrimination. These keys are statisti- 
cally weighted using blind-site ground-truth data to develop a discriminant 
function which Is applied to a large set of segments. 

Personnel of Lockheed Electronics Company, Inc., began work on the LIST proce- 
dure In 1977 in support of the Large Area Crop Inventory Experiment (LACIE) 
within the Earth Observations Division (EOD) at the National Aeronautics and 
Space Administration (NASA), Lyndon B. Johnson Space Center (JSC). The pre- 
liminary develo(Mi«nt was analyzed by Pore (ref. 1). From this analysis, a 
semlautomated operational LIST was developed by Abotteen and Pore (ref. 2) and 
tested on blind sites In both Kansas and North Dakota (ref. 3). Improvements 
In the green number and brightness keys used In LIST were made by Dennis and 
Pore (ref. 4), and the resulting discriminant was tested on Large Area Crop 
Inventory Experiment (LACIE) Phase III Kansas data (ref. 5). In addition, 
alternative spectral keys were studied by Dennis and Pore (ref. 6) for pos- 
sible Inclusion In LIST. 

The Kansas test (ref. 5) demonstrated that LIST could be a useful tool in 
Identifying problem pixels In the labeling process. In view of these positive 
results, a decision was made to transfer the LIST technology to the NASA/JSC/ 
EM) Crop Applications Branch for testing in the fiscal year (FY) 1980 pilot 
test under the ongoing Agriculture and Resources Inventory Surveys Through 
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Remota Sensing (A^ISTARS) program. To transfer a fully developed discrim- 
inant for use on 1979 spring-wheat data. It was necessary to denonstrate the 
year-to-year and geographic extendablllty of LIST. To show this. North Dakota 
data from Phase III (1977) and North Dakota. South Dakota, and Hinnesota data 
from the 1978 Transition Year (TY) were analyzed. The analysis of this data 
Is the subject of this report. 



2. m PROBLEH 


To demonstrati both the temporal and geographic extendablllty of the LIST pro- 
cedure before preceding to the FY 1980 pilot test, the decision Mas made to 
train the LIST discriminant using all available Phase III North Dakota seg- 
ments* A test Mas tiMn made to determine the accuracy of the discriminant on 
the TY North Dakota segnwnts for an Indication of temporal extendablllty. An 
additional test «ms performed on the TY Ninnesota and South Dakota se^nents 
for an Indication of temporal and geographic extendablllty. A complete list 
of the blind-site acquisitions used Is shOMn In table 2-1. 

The training accuracy observed In the Phase III ftorth Dakota segments Mas com- 
parable to the accuracy d>ta1ned In the Kansas test of LIST (ref. 5). These 
training results are presented In table 2-2, along Mith comparative M V>be1- 
Ing accuracy. When the Initial results of applying LIST to the TY data Mere 
presented to the AI's for resolution of the discrepancies. It Mas determined 
that the large number of discrepancies Involved Mould prevent LIST from being 
a useful tool for Identifying problem pixels. 

Test accuracies Mere obtained for the TY data and are presented In table 2-3. 
Note that the accuracy Mas not Improved by considering only the temporal 
extendablllty to the North Dakota TY blind sites. The randomness of these 
results Ci.e., a LIST probability of correct classification (PCC) close to 
50 percent, the high average number of disagreements betMeen LIST and the AI, 
and the lack of 1mprove.ient In PCC over AI accuracy Mhen LIST and the AI 
agreed] seemed to Indicate that a problem existed In the TY data. Ihe 
Phase III raM test data had been merged at the Purdue University for Applica- 
tions of Remote Sensing (LARS) from the existing data tapes at that facility. 
The four channel data tapes for the TY segments had not been transferred to 
LARS, and, therefore, It Mas decided to merge the TY data on the Earth 
Resources Interactive Processing System (ERIPS) and transfer the merged 16- 
channel data to LARS for LIST processing. However, the ERIPS tiwrge was shovm 
not to be random data. Thus, to exclude the possibility of having Improperly 
merged the TY data, a new discriminant was trained using that data. The high 
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TABLE 2-1. • BLIND SITES, LOCATION OF BLIND SIHS, AND ACQUISITIONS 


StgMiit 

County 

Acquisitions 


Phatt III, 

North Dakota 

1602 

Hontrall 

77125, 77143, 77179. 77198 

1606 

Ward 

77125, 77143, 77197, 77250 

1616 

Cavalltr 

77123, 77141, 77159, 77230 

1619 

Grand Forks 

77122, 77140, 77158, 77175 

1622 

Ramay 

77141, 77159, 77176, 77230 

1625 

Dunn 

77125, 77179, 77197, 77233 

1635 

Sheridan 

77105, 77123, 77159, 77195 

1640 

8am«s 

77121, 77140, 77175, 77211 

1648 

BoMHan 

77107, 77125, 77143, 77179 

1652 

Stark 

77125, 77179, 77197, 77233 

1899 

Walsh 

77122, 77157, 77175, 77193 

1902 

McKtnsIt 

77107, 77125, 77144, 77197 

1903 

Mercer 

77125, 77179, 77197, 77233 

1913 

Hettinger 

77125, 77161, 77179, 77215 

TV, North Oakoti 

1394 

Burke 

78120, 78174, 78228, 78264 | 

1457 

Ward 

78174, 78228, 78246, 78264 

1461 

Pierce 

78137, 78190, 78217, 78236 

1472 

Barnes 

78117, 78135, 78216, 78243 

1473 

Cass 

78116, 78197, 78207, 78251 

1584 

Pen4)1na 

78117, 78198, 78216, 78243 

1602 

Montrlal 

78174, 78211, 70228, 78264 

1612 

McHenry 

78137, 78155, 78199, 78236 

1619 

Grand Forks 

78135, 78207, 78243, 70252 

1636 

Stutsman 

78136, 78154, 78217, 78243 

1650 

Hettinger 

78156, 78209, 78218, 78246 

1656 

Morton 

78137, 78155, 78209, 78263 

1658 

Dickey 

78117, 70135, 73207, 78252 

1909 

Kidder 

78136, 78154, 78208, 78217 

1918 

Grant 

78137, 78209, 78236, 78263 

TV, South Dakota 

1668 

Perkins 

78156, 78174, 73228. 78264 

1676 

Bruce 

78135, 78207, 78224, 78234 

1755 

Jerauld 

78117, 78153, 73197, 78225 

1784 

Minnehaha 

78134, 78169, 73196, 78223 

TV, Minnesota 

1380 

Rockwood 

73115, 781G9, 73204, 78222 

1518 

Roseau 

78155. 78130. 70224, 73243 

1566 

Grant 

78133, 78169, 73196. 78232 

1825 

Norman 

73133, 78169. 73196, 70232 

1842 

Yellow Medicine 

78133. 78205, 70223. 78241 
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TABU 2-2.- TRAININ6 RESW.TS FOR PHASE III NORTH DAKOTA SEGHEHTS 
(a) Distribution of LIST labols 



LIST label 

Ground-truth 

label 

Small grains 

Nonsmall grains 

Obvious nonagriculture 

Small grains 

534 

167 

13 

Non^ll grains 

143 

669 

496 


Statistics: 

PCC - 84.07S 
Onisslon rate ■ 25.21% 

Cawlssion rate ■ 10.93% 

Bias - -1.8% 

Average PCC across segments ■ 84.31% 
StaiHlard devlatiwi of PCC «* 4.69% 

PCC, given LIST and AI agree ■ 88.03% 
PCC of LIST on disagreements ■ 40.97% 


(b) Distribution of AI labels 


Ground-truth 

label 

AI label 

Sdisll grains 

Nonsmall grains 

(N>v1ous nonagriculture 

Small grains 

370 

330 

13 

Nonsmall grains 

63 

751 

496 


Statistics: 

PCC - 80.00% 

Omission rate ■ 48.11% 

Commission rate « 4.66% 

Bias - -14.0% 

Average PCC across segments - 80.46% 
Standard devlatlcm of ^CC ■ 9.75% 
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TABLE 2-3.- INITIAL RESULTS FROM aASSIFYING TY DATA 
WITH THE PHASE III DISCRIMINANT 


(a) Distribution of LIST labels for 19 TY sites In 
North Dakota, South Dakota, and Minnesota 


Ground-truth 

label 

LIST label 

Small grains 

Nonsmall grains 

Obvious nonagriculture 

Small grains 

339 

612 

12 

Nonsmall grains 

660 

1005 

246 


Statistics: 

PCC • 55.32% 

Omisslw) rate > 64.(K)% 

Commission rate ■ 34.54% 

Bias - *1.7% 

Average PCC across se^nents ■ 57.13% 
Standard deviation of PCC ■ 20.14% 
PCC, given LIST and A! agree ■ 81.07% 
PCC of LIST on disagreements ■ 18.79% 


(b) Distribution of LIST labels for 
14 North Dakota TY blind sites 


Ground-truth 

label 

LIST label 

Small grains 

Nonsmall grains 

Obvious nonagriculture 

Small grains 

286 

512 

9 

Nonsmall grains 

406 

797 

110 


Statistics: 

PCC • 52.26% 

Omission rate • 63.44% 
Commission rate ■ 30.97% 
Bias > -5.0% 





















training accuracy* shown In table 2>4* ruled out the possibility that raw data 
problems created the random effect shown In table 2>3. A correlation was com- 
puted i^t»^n the two discriminant vectors. 

This correlation shc»^ the vectors to be nearly orthogonal, which Indicates 
that the decision boundary between grains and nongrains changed drastically 
from Phase III to TY. Clearly, there is a real problem In extending the LIST 
discriminant from one year to the next, although the LIST questions still 
appear to discriminate small-grain pixels from other pixels. Some possible 
causes of this poor temporal extendablllty are discussed In the next section. 


TABLE 2-4.- TRAINING RESULTS FOR TY NORTH DAKOTA, 
SOUTH DAKOTA. AND MINNESOTA DATA 



(a) Distribution of LIST labels for 
15 North Dakota blind sites 


Ground-truth 
1 abel 

LIST label 

Small grains 

Nonsmall grains 

Obvious nonagriculture 

Small grains 

502 

323 

10 

Nonsmall grains 

128 

1230 

196 


Statistics: 

PCC = 80.70* 

Omission rate = 39.80% 

Commission rate * 8.20* 

Bias = -8.5* 

Average PCC across segments * 79.23* 
Standard deviation of PCC * 11.22* 
PCC, given LIST and AI agree = 83.6* 
PCC of LIST on Disagreements = 54.1* 


(b) Distribution of LIST labels for 21 North Dakota, 
South Dakota, and Minnesota blind sites 


Ground-truth 

label 

LIST label 

Small grains 

Nonsmall grains 

Obvious nonagriculture 

Small grains 

583 

418 

14 

Nonsmall grains 

127 

1788 

322 


Statistics: 

PCC * 82.81* 

Omission rate = 42.56* 

Commission rate = 5.68* 

Bias = -9.4* 

Average PCC across segments = 84.27* 
Standard deviation of PCC = 11.5* 









3. POSSIBLE CAUSES OF POOR TEMPORAL EXTENDABILITY 


Four factors could have caused or contributed to the lack of temporal 
extendablllty discussed In the previous section. Briefly, these factors are 
as follows: 

a* Inherent difference In the separability of the data from one year to the 
next 

b. A need for temporally current keys 

c. Insufficient representation of AI In the Phase III test 

d. Insufficient training data 

The first factor relates to changes In the separability of the data from 
Phase III to TY. In J. Clinton's study on Labeling characterization (ref. 7), 
the labeling accuracy for North Dakota was shown to be 85.85 percent for 
Phase III segments and 83.94 percent for the TY segments. This Indicates 
that, to a snail degree, the Phase III data are naturally more separable. 

Thus, more latitude is available in the placement of the discriminant bound- 
aries in the Phase III data than in the TY data. Furthermore, when applied to 
the Phase III data, the TY discriminant yields results which are no longer 
random, though they are marginal at best (see table 3-1). 

The second factor relates to the way in which spectral data keys are developed 
and used in LIST. For this study, the four observations of greenness and 
brightness for a given pixel were compared to an expected trajectory of small 
grains for the observed growth stages reported for the segment. This expected 
trajectory was developed using Phase III North Dakota data. Thus, Phase III 
training may not apply to TY small-grain data if, as a whole, the latter data 
follow a different trajectory. Thus, a set of temporally current trajectories 
or temporally current spectral keys may need to be developed. 

The third factor involves a possible imbalance of AI keys in much the same way 
that the second factor involves such an imbalance of spectral keys. 
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TABLE 3-1.- RESULTS FROM CLASSIFYING PHASE III 
DATA WITH THE TY DISCRIMINANT 


[Distribution of LIST labels] 


Ground-truth 

label 

LIST label 

Small grains 

Nonstnall grains 

Obvious nonagriculture 

Small grains 

451 

250 

13 

Nonsmall grains 

385 

427 

496 


Statistics: 

PCC » 67.99% 

Omission rate * 36.83% 

Conmission rate = 29.43% 

Bias * +6.0% 

Average PCC across segments = 67.76% 
Standard deviation of PCC = 21.11% 
PCC, given LIST and AI agree = 83.18% 
PCC of LIST on disagreements = 24.25% 












Tlte problem is that only two AI's were available for answering the LIST 
questions for Phase III data* while for the TY data* 16 AI's provided these 
responses. Two AI's may not provide a broad enough spectrimi of responses to 
obtain a proper balance of those features in the final discriminant. 

Hie final factor relates to the size of the set of training pixels used for 
tte Phase III data. First of all. there were only 14 segments available for 
training purposes in Phase III. Second, the two AI's may have been too 
liberal in assigning pixels the label D, which was intended remove the 
obviously nonagri cultural pixels. In Phase III, 25.07 percent of the total 
pixels available for training were removed because of this designation. By 
comparison, only 10.25 percent of the TY pixels were put in this category. 
Consequently, the effects of this factor and the first factor are difficult, 
if not impossible, to measure. 

In the next section, a method of temporally updating spectral keys is discus- 
sed and applied to the data. Various subsets of the LIST keys are examined 
in the context of their individual contribution to the temporal signature 
extension problem. 
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4. ANALYSIS OF KEYS 

The first attenpt to explain the problem of temporal signature extension was 
to sh(M the Importance of using temporally current spectral keys. Figures 4-1 
ami 4-2 are graphs showing the expected trajectory (given as a mean tl stand- 
ard deviation) of greenness and brightness as a function of the observed 
Robertson biostage. These trajectories were obtained using the available 
ground-truth small-grain pixels from the Phase III North Dakota segments. 

They are used to transform the raw spectral data Into a new set of LIST keys. 

The basic theory Involved Is that the variance of the spectral transformation 
of a pixel from this trajectory Is a measure of the probability of that pixel 
not being a small-grain pixel. However, If meteorological conditions cause 
these trajectories to change from one year to the next, the use of the pre- 
vious year's trajectories may Induce variance which has no relation to the 
true class of the pixels Involved, It was believed that this problem could be 
avoided by using the following operationally feasible method In updating these 1 

trajectories: I 

a. Train the discriminant as before, using one year's worth of data to build \ 

the trajectories and find the weights of the discriminants 

b. Before classifying the next year's data, recompute the trajectories by 
using all the Al-labeled small-grain pixels for that year 


Since AI labels were already provided as part of the AI responses for LIST, no 
new resources were needed to test this concept. A question arises, however. 

In the validity of this approach. Namely, does the AI Induce significant 
errors In the trajectories by being conservative In his labeling. Figures 4-3 
through 4-6 are graphs of the TY greenness and brightness trajectories com- 
puted using ground-truth and AI labels, alternately. These trajectories do 
not appear to be significantly different. Figures 4-7 and 4-8 are greenness 
and brightness trajectories computed using Phase III AI labels. These trajec- 
tories show significant bias when compared to the ground-truth trajectories 
for that year (figures 4-1 and 4-2). Therefore, the Indications are that this 
method of updating keys Is feasible when extending from Phase III to TY, 
although It might not be reliable If the two data sets were reversed. 
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greenness key generated from the ground-truth labels for North Dakota segments 
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e 4-3.— TY greenness key generated from the ground-truth labels. 
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greenness key generated from the AI labels. 



Mean trajectory 



e 4-5.- TY 




Robertson Biostage 

gure 4-6.- TY brightness key generated fro* the AI labels. 
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greenness key generated from the A1 latiels. 
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(^e possible Interpretation Is that the bias noted In the Phase III AI trajec- 
tories Is related to the third factor: only tv« AI's Interpreted all of the 
Phase III segments. 'Dtus, given a broad-based collection of AI's (sii^h as the 
16 AI's used in Interpreting the TY data), the difference In the two methods 
of generating trajectories may not be noticeable. Table 4-1 shows that using 
temporally current trajectories does improve the temporal extendablllty, 
although not significantly. However, the use of temporally current keys In 
approaching the temporal extendablllty problem Is not considered significant. 
It was decided, therefore, that In testing subsets of the keys for extend- 
ablllty. the ground-truth trajectories for Phase III would be used with the 
Phase III data and the Al-generated TY trajectories would be used with the TY 
data. 

With the contribution of temporally current trajectories thus removed from the 
problem, the next most obvious difference In the spectral keys was In the 
change In the brightness trajectory from Phase III to TY, Table 4-2 shows the 
results obtained from removing the brightness keys and trying to extend from 
Phase III to TY using only the AI and greenness variables. These results 
I i reveal that brightness alone Is not the cause of the poor extendablllty. 

Tables 4-3, 4-4, and 4-5 show the extendablllty achieved using greenness/ 

I brightness keys, only greenness keys, and only AI keys, respectively. Table 

‘ 4-6 shows training and Independent test accuracies for those sets of keys 

idilch were applied to the TY data, reiterating that the problem involves only 
i the temporal extandability of LIST. 
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TABLE 4-1.- ACCURACY OF EXTENSION WITH UPDATED KEYS 

(a) Distribution of LIST labels In classification of 24 TY se$nents 
with Phase III trained discriminant and updated keys 


LIST label 


Ground-truth 




label 

Small grains 

Nonsmall grains 

Obvious nonagriculture 

Small grains 

321 

739 

14 

Nonsmall grains 

912 

1593 

359 


Statistics: 

PCC » 57.72* 

Omission rate = 70.11* 

Commission rate « 31.84* 

Bias = +7.45* 

Average PCC across segments * 63* 
Standard deviation of PCC = 18.89* 
PCC, given LIST and AI agree = 84.55* 
PCC of LIST on disagreements = 18.8* 


(b) Distribution of LIST labels in classification of 19 TY segments 
with Phase III weights without updated keys 


Ground-truth 

label 

Small grains 
Nonsmall grains 


LIST label 

Small grains Nonsmall grains Obvious nonagriculture 
339 612 12 


Statistics: 

PCC » 55.32* 

Omission rate = 64* 

Commission rate = 34.54* 

Bias = +1.7* 

Average PCC across segments * 57.13* 
Standard deviation of PCC » 20.14* 
PCC, given LIST and AI agree * 81.07* 
PCC of LIST on Disagreements = 18.79* 
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TABLE 4.2.. RESULTS OBTAINED BY REMOVING BRIGHTNESS KEYS 


[Distribution of LIST labels for 24 TY blind sites* 
classified from Riase III training] 


Ground-tfHitb 

latel 

LIST label 

^11 grains 

Nonsiaall grains 

Obvious nonagriculture 

Small grains 

465 

595 

14 

NonsflMlI grains 

694 

1511 

359 


Statistics: 

PCC » 64.18* 

Omission rate > 55.4% 

Commission rate = 27.07% 

Bias = +2.7% 

Average PCC across segnents = 64.67% 
Standard deviation of PCC = 16.97% 


1 I 
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TABLE 4-3.- RESULTS USING ONLY GREENNESS/BRIGHTNESS KEYS 



Data s?t classified 

Data used 
In training 

Phase III 

TY 

Mean PCC 

Standard 

deviation 

Mean PCC 

Standard 

deviation 

Phase III 

83.78 

5.19 

63.58 

17.6 

TY 

70.26 

17.27 

82.42 

10.26 
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TABLE 4-4.- RESULTS USING ONLY GREENNESS KEYS 



Data set classified 

Data used 

Phase III 

TY 

a n wi fl 1 n 1 ny 

Mean PCC 

Standard 

deviation 

Mean PCC 

Standard 

deviation 

Phase III 

81.89 

8.71 

65.74 

16.87 

TY 

72.62 

20.18 

77.24 

12.8 
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TABLE 4-5.- EXTENDABILITY ACHIEVED USING ANALYST KEYS ONLY 

(a) Results 


Data classified 


(b) Probability of agreement of machine 
classified label and analyst label 
(classified using only AI keys) 


Data used 
to train 
discriminant 


Phase III 
TY 


Data classified 

Phase III 

TY 

0.567 

0.637 

.672 

.871 























TABLE 4-6.- TRAINING AND TEST ACCURACY OF KEYS 
APPLIED TO THE TY DATA 


Data set 

Mean PCC 

Standard deviation 
of the PCC 

Greenness and brightness keys 

Training data 

81.52 

10.30 

Test data 

84.24 

10.63 

Greenness keys only 

Training data 

75.87 

12.62 

Test data 

79.97 

13.56 

AI keys only 

Training data 

72.03 

15.69 

Test data 

76.88 

30.2 

















5. CONCLUSION 


The most apparent contributors to the problem of poor temporal extension of 
LIST are the drastic changes in the brightness keys and an inadequate set of 
AI responses in Phase III. The brightness trajectories change drastically 
from Phase III to the TY. When classifying data, the inclusion of brightness 
channels with greenness channels increases accuracy within a year, but this 
combination decreases accuracy when used in classifying data from a year other 
than that of the training data. This indicates that brightness does not 
follow a trajectory which is consistently related to crop growth over several 
years. 

Removing brightness channels from the discriminant, however, does not com- 
pletely correct the lack of extendability. Note from tables 4-1 and 4-2 that 
removing brightness increases the accuracy of the extension from Phase III to 
TY from 57.72 percent to 64.18 percent. Note also that removal of the AI keys 
increases accuracy to 65.76 percent (table 4-4). Although the latter increase 
appears insignificant when compared to the first, note that removing only the 
AI keys increased accuracy to 63.58 percent (table 4-3). 

Table 4-5 shows that it is the set of Phase III AI responses which is contri- 
buting most to the problem. Note that proper weighting of the responses 
explains 73.8 percent of the ground-truth labels but only 56.7 percent of the 
AI labels. By contrast, the TY responses which were weighted to explain the 
TY ground-truth labels faired equally well, explaining 73.6 percent of those 
labels and 87.1 percent of the AI labels. 
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6. RECOMMENDATIONS 


A satisfactory signature extension from Phase III to the TY has not been 
achieved with the current data and a meaningful subset of the current keys. 

It is possible, however, that with good AI responses for the Phase III data, 
some improvement would be manifested. On the other hand, it is possible that 
the questions asked of AI's cannot be answered reliably. In order to resolve 
these issues, it is recommended that multiple sets of responses to the LIST 
questions be collected for the Phase III North Dakota blind sites using a 
large number of AI's. It is also reconmended that these responses be analyzed 
to determine (1) the consistency with which the LIST questions can be 
answered, (2) the relationship between the responses to Lne LIST questions and 
the AI label, and (3) the role of AI responses in the problem of temporal 
signature extension. 
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